Pandas Profiling Report

Dataset statistics

Number of variables	10
Number of observations	20640
Missing cells	207
Missing cells (%)	0.1%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	1.5 MiB
Average record size in memory	76.0 B

Variable types

NUM	9
CAT	1

Reproduction

Analysis started	2020-06-30 17:55:56.933729
Analysis finished	2020-06-30 17:57:03.187439
Duration	1 minute and 6.25 seconds
Version	pandas-profiling v2.8.0
Command line	`pandas_profiling --config_file config.yaml [YOUR_FILE.csv]`
Download configuration	config.yaml

Warnings

`latitude` is highly correlated with `longitude`	High correlation
`longitude` is highly correlated with `latitude`	High correlation
`total_bedrooms` is highly correlated with `total_rooms` and 1 other fields	High correlation
`total_rooms` is highly correlated with `total_bedrooms` and 1 other fields	High correlation
`households` is highly correlated with `total_rooms` and 2 other fields	High correlation
`population` is highly correlated with `households`	High correlation
`total_bedrooms` has 207 (1.0%) missing values	Missing

longitude
Real number (ℝ)

HIGH CORRELATION

Distinct count	844
Unique (%)	4.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	-119.56970445736432
Minimum	-124.35
Maximum	-114.31
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	-124.35
5-th percentile	-122.47
Q1	-121.8
median	-118.49
Q3	-118.01
95-th percentile	-117.08
Maximum	-114.31
Range	10.04
Interquartile range (IQR)	3.79

Descriptive statistics

Standard deviation	2.003531724
Coefficient of variation (CV)	-0.01675618195
Kurtosis	-1.330152366
Mean	-119.5697045
Median Absolute Deviation (MAD)	1.28
Skewness	-0.297801208
Sum	-2467918.7
Variance	4.014139367

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
-118.31	162	0.8%
-118.3	160	0.8%
-118.29	148	0.7%
-118.27	144	0.7%
-118.32	142	0.7%
-118.28	141	0.7%
-118.35	140	0.7%
-118.36	138	0.7%
-118.19	135	0.7%
-118.25	128	0.6%
Other values (834)	19202	93.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
-124.35	1	< 0.1%
-124.3	2	< 0.1%
-124.27	1	< 0.1%
-124.26	1	< 0.1%
-124.25	1	< 0.1%

Value	Count	Frequency (%)
-114.31	1	< 0.1%
-114.47	1	< 0.1%
-114.49	1	< 0.1%
-114.55	1	< 0.1%
-114.56	1	< 0.1%

latitude
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct count	862
Unique (%)	4.2%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	35.63186143410853
Minimum	32.54
Maximum	41.95
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	32.54
5-th percentile	32.82
Q1	33.93
median	34.26
Q3	37.71
95-th percentile	38.96
Maximum	41.95
Range	9.41
Interquartile range (IQR)	3.78

Descriptive statistics

Standard deviation	2.135952397
Coefficient of variation (CV)	0.05994501302
Kurtosis	-1.117759781
Mean	35.63186143
Median Absolute Deviation (MAD)	1.23
Skewness	0.4659530037
Sum	735441.62
Variance	4.562292644

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
34.06	244	1.2%
34.05	236	1.1%
34.08	234	1.1%
34.07	231	1.1%
34.04	221	1.1%
34.09	212	1.0%
34.02	208	1.0%
34.1	203	1.0%
34.03	193	0.9%
33.93	181	0.9%
Other values (852)	18477	89.5%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
32.54	1	< 0.1%
32.55	3	< 0.1%
32.56	10	< 0.1%
32.57	18	0.1%
32.58	26	0.1%

Value	Count	Frequency (%)
41.95	2	< 0.1%
41.92	1	< 0.1%
41.88	1	< 0.1%
41.86	3	< 0.1%
41.84	1	< 0.1%

housing_median_age
Real number (ℝ_≥0)

Distinct count	52
Unique (%)	0.3%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	28.639486434108527
Minimum	1.0
Maximum	52.0
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	1
5-th percentile	8
Q1	18
median	29
Q3	37
95-th percentile	52
Maximum	52
Range	51
Interquartile range (IQR)	19

Descriptive statistics

Standard deviation	12.58555761
Coefficient of variation (CV)	0.4394477408
Kurtosis	-0.8006288536
Mean	28.63948643
Median Absolute Deviation (MAD)	10
Skewness	0.0603306376
Sum	591119
Variance	158.3962604

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
52	1273	6.2%
36	862	4.2%
35	824	4.0%
16	771	3.7%
17	698	3.4%
34	689	3.3%
26	619	3.0%
33	615	3.0%
18	570	2.8%
25	566	2.7%
Other values (42)	13153	63.7%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1	4	< 0.1%
2	58	0.3%
3	62	0.3%
4	191	0.9%
5	244	1.2%

Value	Count	Frequency (%)
52	1273	6.2%
51	48	0.2%
50	136	0.7%
49	134	0.6%
48	177	0.9%

total_rooms
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct count	5926
Unique (%)	28.7%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	2635.7630813953488
Minimum	2.0
Maximum	39320.0
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	2
5-th percentile	620.95
Q1	1447.75
median	2127
Q3	3148
95-th percentile	6213.2
Maximum	39320
Range	39318
Interquartile range (IQR)	1700.25

Descriptive statistics

Standard deviation	2181.615252
Coefficient of variation (CV)	0.8276977802
Kurtosis	32.630927
Mean	2635.763081
Median Absolute Deviation (MAD)	797
Skewness	4.147343451
Sum	54402150
Variance	4759445.106

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
1527	18	0.1%
1613	17	0.1%
1582	17	0.1%
2127	16	0.1%
1703	15	0.1%
1471	15	0.1%
2053	15	0.1%
1722	15	0.1%
1607	15	0.1%
1717	15	0.1%
Other values (5916)	20482	99.2%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
2	1	< 0.1%
6	1	< 0.1%
8	1	< 0.1%
11	1	< 0.1%
12	1	< 0.1%

Value	Count	Frequency (%)
39320	1	< 0.1%
37937	1	< 0.1%
32627	1	< 0.1%
32054	1	< 0.1%
30450	1	< 0.1%

total_bedrooms
Real number (ℝ_≥0)

HIGH CORRELATION
MISSING

Distinct count	1923
Unique (%)	9.4%
Missing	207
Missing (%)	1.0%
Infinite	0
Infinite (%)	0.0%

Mean	537.8705525375618
Minimum	1.0
Maximum	6445.0
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	1
5-th percentile	137
Q1	296
median	435
Q3	647
95-th percentile	1275.4
Maximum	6445
Range	6444
Interquartile range (IQR)	351

Descriptive statistics

Standard deviation	421.3850701
Coefficient of variation (CV)	0.7834321252
Kurtosis	21.98557506
Mean	537.8705525
Median Absolute Deviation (MAD)	162
Skewness	3.459546332
Sum	10990309
Variance	177565.3773

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
280	55	0.3%
331	51	0.2%
345	50	0.2%
393	49	0.2%
343	49	0.2%
394	48	0.2%
328	48	0.2%
348	48	0.2%
272	47	0.2%
309	47	0.2%
Other values (1913)	19941	96.6%
(Missing)	207	1.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1	1	< 0.1%
2	2	< 0.1%
3	5	< 0.1%
4	7	< 0.1%
5	6	< 0.1%

Value	Count	Frequency (%)
6445	1	< 0.1%
6210	1	< 0.1%
5471	1	< 0.1%
5419	1	< 0.1%
5290	1	< 0.1%

population
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct count	3888
Unique (%)	18.8%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	1425.4767441860465
Minimum	3.0
Maximum	35682.0
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	3
5-th percentile	348
Q1	787
median	1166
Q3	1725
95-th percentile	3288
Maximum	35682
Range	35679
Interquartile range (IQR)	938

Descriptive statistics

Standard deviation	1132.462122
Coefficient of variation (CV)	0.7944444737
Kurtosis	73.55311639
Mean	1425.476744
Median Absolute Deviation (MAD)	440
Skewness	4.935858227
Sum	29421840
Variance	1282470.457

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
891	25	0.1%
761	24	0.1%
1227	24	0.1%
850	24	0.1%
1052	24	0.1%
825	23	0.1%
999	22	0.1%
782	22	0.1%
1005	22	0.1%
781	21	0.1%
Other values (3878)	20409	98.9%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
3	1	< 0.1%
5	1	< 0.1%
6	1	< 0.1%
8	4	< 0.1%
9	2	< 0.1%

Value	Count	Frequency (%)
35682	1	< 0.1%
28566	1	< 0.1%
16305	1	< 0.1%
16122	1	< 0.1%
15507	1	< 0.1%

households
Real number (ℝ_≥0)

HIGH CORRELATION

Distinct count	1815
Unique (%)	8.8%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	499.5396802325581
Minimum	1.0
Maximum	6082.0
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	1
5-th percentile	125
Q1	280
median	409
Q3	605
95-th percentile	1162
Maximum	6082
Range	6081
Interquartile range (IQR)	325

Descriptive statistics

Standard deviation	382.3297528
Coefficient of variation (CV)	0.7653641301
Kurtosis	22.05798806
Mean	499.5396802
Median Absolute Deviation (MAD)	151
Skewness	3.410437712
Sum	10310499
Variance	146176.0399

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
306	57	0.3%
386	56	0.3%
335	56	0.3%
282	55	0.3%
429	54	0.3%
375	53	0.3%
284	51	0.2%
297	51	0.2%
362	50	0.2%
380	50	0.2%
Other values (1805)	20107	97.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
1	1	< 0.1%
2	3	< 0.1%
3	4	< 0.1%
4	4	< 0.1%
5	7	< 0.1%

Value	Count	Frequency (%)
6082	1	< 0.1%
5358	1	< 0.1%
5189	1	< 0.1%
5050	1	< 0.1%
4930	1	< 0.1%

median_income
Real number (ℝ_≥0)

Distinct count	12928
Unique (%)	62.6%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	3.8706710029069766
Minimum	0.4999
Maximum	15.0001
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	0.4999
5-th percentile	1.60057
Q1	2.5634
median	3.5348
Q3	4.74325
95-th percentile	7.300305
Maximum	15.0001
Range	14.5002
Interquartile range (IQR)	2.17985

Descriptive statistics

Standard deviation	1.899821718
Coefficient of variation (CV)	0.4908249026
Kurtosis	4.952524102
Mean	3.870671003
Median Absolute Deviation (MAD)	1.0642
Skewness	1.646656702
Sum	79890.6495
Variance	3.60932256

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
3.125	49	0.2%
15.0001	49	0.2%
2.875	46	0.2%
4.125	44	0.2%
2.625	44	0.2%
3.875	41	0.2%
3	38	0.2%
3.375	38	0.2%
3.625	37	0.2%
4	37	0.2%
Other values (12918)	20217	98.0%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
0.4999	12	0.1%
0.536	10	< 0.1%
0.5495	1	< 0.1%
0.6433	1	< 0.1%
0.6775	1	< 0.1%

Value	Count	Frequency (%)
15.0001	49	0.2%
15	2	< 0.1%
14.9009	1	< 0.1%
14.5833	1	< 0.1%
14.4219	1	< 0.1%

median_house_value
Real number (ℝ_≥0)

Distinct count	3842
Unique (%)	18.6%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%

Mean	206855.81690891474
Minimum	14999.0
Maximum	500001.0
Zeros	0
Zeros (%)	0.0%
Memory size	161.2 KiB

Quantile statistics

Minimum	14999
5-th percentile	66200
Q1	119600
median	179700
Q3	264725
95-th percentile	489810
Maximum	500001
Range	485002
Interquartile range (IQR)	145125

Descriptive statistics

Standard deviation	115395.6159
Coefficient of variation (CV)	0.55785531
Kurtosis	0.3278702429
Mean	206855.8169
Median Absolute Deviation (MAD)	68400
Skewness	0.9777632739
Sum	4269504061
Variance	1.331614816e+10

Histogram

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
500001	965	4.7%
137500	122	0.6%
162500	117	0.6%
112500	103	0.5%
187500	93	0.5%
225000	92	0.4%
350000	79	0.4%
87500	78	0.4%
275000	65	0.3%
150000	64	0.3%
Other values (3832)	18862	91.4%

Minimum 5 values
Maximum 5 values

Value	Count	Frequency (%)
14999	4	< 0.1%
17500	1	< 0.1%
22500	4	< 0.1%
25000	1	< 0.1%
26600	1	< 0.1%

Value	Count	Frequency (%)
500001	965	4.7%
500000	27	0.1%
499100	1	< 0.1%
499000	1	< 0.1%
498800	1	< 0.1%

ocean_proximity
Categorical

Distinct count	5
Unique (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	80.6 KiB

<1H OCEAN	9136
INLAND	6551
NEAR OCEAN	2658
NEAR BAY	2290
ISLAND	5

Common Values
Length

Value	Count	Frequency (%)
<1H OCEAN	9136	44.3%
INLAND	6551	31.7%
NEAR OCEAN	2658	12.9%
NEAR BAY	2290	11.1%
ISLAND	5	< 0.1%

Length

Max length	10
Median length	9
Mean length	8.064922481
Min length	6

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

First rows

	longitude	latitude	housing_median_age	total_rooms	total_bedrooms	population	households	median_income	median_house_value	ocean_proximity
0	-122.23	37.88	41.0	880.0	129.0	322.0	126.0	8.3252	452600.0	NEAR BAY
1	-122.22	37.86	21.0	7099.0	1106.0	2401.0	1138.0	8.3014	358500.0	NEAR BAY
2	-122.24	37.85	52.0	1467.0	190.0	496.0	177.0	7.2574	352100.0	NEAR BAY
3	-122.25	37.85	52.0	1274.0	235.0	558.0	219.0	5.6431	341300.0	NEAR BAY
4	-122.25	37.85	52.0	1627.0	280.0	565.0	259.0	3.8462	342200.0	NEAR BAY
5	-122.25	37.85	52.0	919.0	213.0	413.0	193.0	4.0368	269700.0	NEAR BAY
6	-122.25	37.84	52.0	2535.0	489.0	1094.0	514.0	3.6591	299200.0	NEAR BAY
7	-122.25	37.84	52.0	3104.0	687.0	1157.0	647.0	3.1200	241400.0	NEAR BAY
8	-122.26	37.84	42.0	2555.0	665.0	1206.0	595.0	2.0804	226700.0	NEAR BAY
9	-122.25	37.84	52.0	3549.0	707.0	1551.0	714.0	3.6912	261100.0	NEAR BAY

Last rows

	longitude	latitude	housing_median_age	total_rooms	total_bedrooms	population	households	median_income	median_house_value	ocean_proximity
20630	-121.32	39.29	11.0	2640.0	505.0	1257.0	445.0	3.5673	112000.0	INLAND
20631	-121.40	39.33	15.0	2655.0	493.0	1200.0	432.0	3.5179	107200.0	INLAND
20632	-121.45	39.26	15.0	2319.0	416.0	1047.0	385.0	3.1250	115600.0	INLAND
20633	-121.53	39.19	27.0	2080.0	412.0	1082.0	382.0	2.5495	98300.0	INLAND
20634	-121.56	39.27	28.0	2332.0	395.0	1041.0	344.0	3.7125	116800.0	INLAND
20635	-121.09	39.48	25.0	1665.0	374.0	845.0	330.0	1.5603	78100.0	INLAND
20636	-121.21	39.49	18.0	697.0	150.0	356.0	114.0	2.5568	77100.0	INLAND
20637	-121.22	39.43	17.0	2254.0	485.0	1007.0	433.0	1.7000	92300.0	INLAND
20638	-121.32	39.43	18.0	1860.0	409.0	741.0	349.0	1.8672	84700.0	INLAND
20639	-121.24	39.37	16.0	2785.0	616.0	1387.0	530.0	2.3886	89400.0	INLAND

Overview

Variables

Interactions

Correlations

Pearson's r

Spearman's ρ

Kendall's τ

Phik (φk)

Missing values

Sample

First rows

Last rows